An Empirical Analysis of Proximal Policy Optimization with Kronecker-factored Natural Gradients

Deep reinforcement learning methods have shown tremendous success in a large variety tasks, such as Go [Silver et al., 2016], Atari [Mnih et al., 2013], and continuous control [Lillicrap et al., 2015, Schulman et al., 2015]. Policy gradient methods [Williams, 1992] is an important family of methods in model-free reinforcement learning, and the current state-of-the-art policy gradient methods are Proximal Policy Optimization ( Schulman et al. [2017]) and ACKTR [Wu et al., 2017]. The two methods, however, take different approaches to better sample efficiency: PPO considers a particular “clipping” objective that mimics a trust-region, whereas ACKTR considers approximated natural gradients that balances speed and optimization.